Search AI Products and News

DiPIR is an innovative technology designed to address the technical challenges of seamlessly inserting virtual objects into real scenes. It achieves this by combining large-scale diffusion models with physics-based inverse rendering processes to accurately recover scene lighting information from a single image, allowing for the natural integration of virtual objects with their environments. The workflow of DiPIR includes constructing virtual 3D scenes, simulating interactions between objects and environments, and optimizing lighting conditions through diffusion models to ensure that generated images conform to real scenes. Its advantages include wide applicability, suitable for scenes under different lighting conditions, and the generation of highly realistic images.

The research team proposed a simpler approach, IllumiNeRF: first, a diffusion model under specific lighting conditions is used to relight each input image, and then these relighted images are used to reconstruct a Neural Radiance Field (NeRF) to present new perspectives under the target lighting. They demonstrated that this strategy is surprisingly competitive and achieved state-of-the-art results in several relighting benchmarks.